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ABSTRACT 


Acquisition  costs  of  Department  of  Defense  (DoD)  weapon  systems  are  of  major 
concern  due  to  their  enormous  annual  expense.  In  fact,  $165B  was  spent  by 
DoD  on  acquisition  in  2008.  However,  another  budget  component,  Product 
Support,  costs  the  government  an  additional  $132B  annually.  DoD  has  stated 
that  the  preferred  method  for  providing  this  life-cycle  sustainment  is  Performance 
Based  Logistics  (PBL),  and  20%  of  all  programs  now  use  that  method  of  support. 
PBL’s  goal  is  to  provide  maximum  readiness  at  reasonable  costs. 

Unfortunately,  evaluation  of  the  success  of  these  PBL  contracts  is  difficult, 
because  of  the  large  number  of  metrics  and  vast  differences  among  weapon 
systems.  Several  studies  have  attempted  to  do  so,  but  they  have  not  used  DoD- 
prescribed  metrics  for  evaluation.  This  research  attempts  to  use  two  of  the  five 
DoD-prescribed  metrics  to  analyze  the  PBL  contract  for  the  Army  High  Mobility 
Medium  Wheeled  Vehicle  (HMMWV)  Integrated  Logistics  Plan  (ILP).  These  two 
are  Cost  Per  Unit  Usage  (CPUU)  and  Operational  Availability. 

This  statistical  analysis  found  that  the  ILP  does  not  appear  to  be 
decreasing  cost  or  increasing  Operational  Availability  for  the  Army  HMMWV 
program.  Further  study  using  the  other  three  DoD  metrics  is  recommended. 
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EXECUTIVE  SUMMARY 


Acquisition  costs  of  DoD  weapon  systems  are  of  major  concern  due  to  their 
enormous  annual  expense.  In  fact,  $165B  was  spent  by  Department  of  Defense 
(DoD)  on  acquisition  in  2008.  However,  another  budget  component,  Product 
Support,  costs  the  government  an  additional  $132B  annually.  DoD  has  stated 
that  the  preferred  method  for  providing  this  life-cycle  sustainment  is  Performance 
Based  Logistics  (PBL),  and  20%  of  all  programs  now  use  that  method  of  support. 
PBL’s  goal  is  to  provide  maximum  readiness  at  reasonable  costs. 

In  October  2009,  President  Obama  signed  the  Fiscal  Year  2010  National 
Defense  Authorization  Act.  This  act  contained  a  provision  entitled  "Life-cycle 
Management  and  Product  Support"  requiring  that:  (1)  the  Secretary  of  Defense 
issue  comprehensive  guidance  on  Life-cycle  Management  (LCM),  and  on  the 
development  and  implementation  of  product  support  strategies  for  major  weapon 
systems;  (2)  each  major  weapons  system  be  supported  by  a  Product  Support 
Manager  (PSM);  and  (3)  each  PSM  position  be  performed  by  a  properly  qualified 
member  of  the  armed  forces  or  full-time  employee  of  the  DoD.  Product  support 
and  sustainment  thus  became  a  high  priority  for  DoD. 

Since  1999,  the  DoD  has  been  transforming  to  a  system  that  places  more 
emphasis  on  readiness  outcomes.  Performance-Based  Logistics  (PBL)  has  been 
named  the  preferred  method  of  life-cycle  sustainment.  PBL  has  allowed 
partnerships  between  government  and  industry  to  be  formed.  These 
partnerships  are  designed  to  bring  together  the  best  of  the  public  and  private 
sectors  in  order  to  provide  DoD  with  maximum  readiness  at  reasonable  costs. 

The  Army  has  been  working  through  this  transformation  in  life-cycle 
sustainment.  Specifically  in  2006,  it  implemented  an  Integrated  Logistics  Plan  for 
the  recapitalization  (RECAP)  program  for  High  Mobility  Medium  Wheeled  Vehicle 
(HMMWV)  vehicles.  This  created  a  partnership  with  AM  General  to  provide  parts 
to  Army  Depots  conducting  the  RECAP  of  HMMWVs.  This  study  analyzes  the 
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Cost  Per  Unit  Usage  (CPUU)  and  Operational  Availability  (A0)  of  the  Army 
HMMWV.  The  CPUU  and  Operational  Availability  were  analyzed  over  a  six  year 
period  from  2004  to  2009. 

The  primary  goal  of  this  thesis  was  to  conduct  a  comparison  of  different 
support  strategies  on  a  weapon  system.  Data  was  collected  from  the  Army 
Logistics  Integrated  Database  (LIDB)  and  Operating  and  Support  Management 
Information  System  (OSMIS).  This  study  focused  on  the  M998  and  Ml 097 
models,  which  are  two  of  the  models  that  went  through  the  RECAP  process.  The 
data  for  the  usage  (mileage)  of  the  HMMWV’s  contained  numerous  anomalies 
and  was  missing  numerous  odometer  readings.  This  made  computing  the  CPUU 
difficult,  as  we  had  to  make  several  corrections  to  the  usage  data  under 
assumptions  that  we  determined  to  be  reasonable  based  on  the  usable  data. 

Ultimately,  we  found  that  the  ILP  does  not  appear  to  be  providing  any 
costs  savings  or  increased  Operational  Availability  since  its  implementation.  The 
CPUU  had  an  upward  trend  over  the  six-year  period  and  the  pre-ILP  cost  was 
found  to  be  cheaper  than  the  post-ILP  cost.  The  Operational  Availability  was 
consistent  at  90%  to  95%  across  this  same  time  period.  The  major  finding  in  this 
study  was  that  the  data  available  to  conduct  these  type  of  studies  is  poorly 
maintained.  The  amount  of  missing  and  anomalous  data  does  not  allow  analysts 
to  confidently  draw  conclusions.  It  is  recommended  that  DoD  invest  time  and 
money  in  data  collection  and  database  organization  in  order  to  provide  analysts 
with  accurate  and  complete  data. 

Studies  such  as  this  can  provide  valuable  information  to  key  decision 
makers  within  the  DoD.  However,  in  order  to  provide  that  valuable  information, 
analysts  need  to  be  provided  with  complete  and  accurate  data.  Providing  our 
warfighters  with  the  best  weapons  systems  at  reasonable  costs  to  the  taxpayer  is 
a  key  issue  in  the  United  States.  It  is  critical  that  DoD  analysts  be  provided  with 
cost  and  availability  information  to  allow  them  to  make  accurate  estimates  of 
weapons  system  support. 
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I.  INTRODUCTION 


Acquisition  costs  of  DoD  weapon  systems  are  of  major  concern  due  to 
their  enormous  annual  expense.  In  fact,  $165B  was  spent  by  DoD  on  acquisition 
in  2008.  However,  another  budget  component,  Product  Support,  costs  the 
government  an  additional  $132B  annually.  (U.S.  Department  of  Defense,  Deputy 
Under  Secretary  of  Defense  for  Logistics  and  Materiel  Readiness,  2009,  p.  6) 
DoD  has  stated  that  the  preferred  method  for  providing  this  life-cycle  sustainment 
is  Performance  Based  Logistics  (PBL),  and  20%  of  all  programs  now  use  that 
method  of  support.  PBL’s  goal  is  to  provide  maximum  readiness  at  reasonable 
costs. 

For  years  product  managers  managed  components  of  readiness  as 
discrete  units.  In  other  words,  they  bought  and  stocked  parts,  systems,  and  sub¬ 
systems,  and  shipped  them  to  units  as  necessary.  Since  1999,  the  DoD  has 
been  evolving  to  a  system  that  places  more  emphasis  on  readiness  outcomes. 
Performance-Based  Logistics  (PBL)  is  the  “poster  child”  of  this  strategy.  PBL 
has  allowed  partnerships  between  government  and  industry  to  be  formed.  These 
partnerships  were  designed  to  bring  together  the  best  of  the  public  and  private 
sectors  in  order  to  provide  DoD  with  maximum  readiness  at  reasonable  costs. 
While  this  strategy  has  been  declared  as  the  preferred  means  of  life-cycle 
support  (U.S.  Department  of  Defense,  2007,  p.  7),  fewer  than  20%  of  all 
weapons  systems  are  using  this  strategy.  PBL  has  been  criticized  as  simply 
being  “contracted  logistics.”  Many  have  questioned  the  cost-effectiveness  of  this 
strategy.  The  Government  Accountability  Office  (GAO)  has  conducted  two 
separate  studies  on  PBL  and  has  yet  to  find  any  conclusive  evidence  that  PBL 
contracts  are  saving  the  government  money.  Other  studies  have  indicated 
increased  levels  of  availability.  Fewer  people  have  raised  questions  concerning 
this  metric  than  with  cost.  Lawmakers,  DoD  administrators,  and  our  Military 
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Commanders  need  to  be  able  to  look  at  the  costs  and  availability  together  to 
determine  if  these  types  of  support  strategies  are  the  future  of  weapon  systems 
life-cycle  sustainment. 

A.  BACKGROUND 

In  2005,  an  Integrated  Logistics  Plan  (ILP)  was  developed  in  an  attempt  to 
increase  readiness  for  the  Army  HMMWV.  Two  production  lines,  Letterkenny 
Army  Depot  (LEAD)  in  Franklin  County,  Pennsylvania,  and  Red  River  Army 
Depot  (RRAD)  in  Bowie  County,  Texas,  were  chosen  for  the  program.  This  was 
expanded  to  include  Maine  Military  Authority  (MMA)  in  Aroostook  County,  Maine, 
in  May  of  2006.  (Smith,  2009,  p.  6)  The  idea  behind  the  ILP  was  to  move  the 
“point  of  sale”  to  the  “point  of  use”  by  having  the  supplier  deliver  the  parts  and 
having  minimal  government  intervention  in  the  supply  chain.  AM  General  was 
awarded  a  $200  million  contract  to  provide  parts  for  the  HMMWV  recapitalization 
program  at  the  two  Army  Depots. 

The  overall  strategy  of  the  program  was  to  optimize  support  to  the 
warfighter,  which  is  also  the  overall  goal  of  the  DoD  policy  on  weapons  system 
support.  Additionally,  this  partnership  was  expected  to  save  the  Army  money  on 
Operations  and  Support  (O&S)  costs.  The  contract  with  AM  General  was 
renewed  in  2008.  The  Army  has  reported  an  overall  savings  of  $4,520  and 
$3,414  per  vehicle  at  LEAD  and  RRAD,  respectively.  Considering  the  large 
number  of  HMMWV’s  the  Army  has  in  its  inventory,  this  is  a  substantial  savings 
to  the  DoD.  Army  Material  Systems  Analysis  Activity  (AMSAA)  reported  an 
overall  savings  of  over  $800,000  for  FY06.  (Chapman,  2007) 

The  Army  and  other  services  continue  to  modify  their  maintenance 
strategies  to  try  and  find  ways  to  increase  availability  and  save  costs.  Each  of 
these  strategies  needs  to  be  analyzed  to  determine  effects  on  costs  and 
availability  and  to  give  decision  makers  a  basis  for  their  future  plans. 
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B.  ANALYSIS 


The  key  metrics  of  this  study  were  Cost  Per  Unit  Usage  and  Operational 
Availability.  The  years  2004-2005  were  used  as  a  baseline,  since  these  were 
the  two  years  of  data  prior  to  the  ILP  being  put  in  place.  Then  the  years  2006- 
2009  were  compared  to  the  baseline  to  determine  if  there  were  any  costs  savings 
or  increased  Operational  Availability.  We  also  looked  at  the  trends  across  the 
entire  time  frame  to  determine  what  the  CPUU  and  Operational  Availability  were 
doing  over  time.  All  the  analysis  was  conducted  in  R  (R  Development  Core 
Team,  2010)  and  Microsoft  Excel  (Excel,  2007). 

The  largest  portion  of  our  time  was  spent  handling  the  data.  In  particular, 
the  usage  data  required  substantial  repairs  due  to  missing  and  anomalous  data. 
The  serial  numbers  for  the  vehicles  also  required  several  repairs  mostly  due  to 
obvious  entry  errors.  Once  the  data  was  corrected  and  put  in  a  usable  format, 
we  used  several  different  strategies  to  examine  the  effects  of  the  ILP.  For 
instance,  the  metrics  were  categorized  by  Major  Army  Command  (MACOM)  to 
eliminate  the  possibility  that  some  of  the  smaller  commands  had  too  much 
leverage  on  the  data.  We  used  statistical  tests  to  determine  if  the  differences 
between  the  time  periods  were  statistically  significant.  The  data  was  broken 
down  by  quarters  and  analyzed  to  determine  trends  over  the  course  of  the  study 
period.  Ultimately,  all  the  analyses  led  to  similar  conclusions. 

C.  SUMMARY  OF  FINDINGS 

The  Cost  Per  Unit  Usage  (CPUU)  was  found  to  have  a  rising  trend  over 
the  time  period  studied.  When  comparing  the  two  time  periods,  pre-ILP  and 
post-ILP,  the  study  determined  that  the  pre-ILP  CPUU  was  less  expensive. 
Several  different  analyses  were  conducted  and  all  of  them  came  to  the  same  two 
conclusions:  the  Army  HMMWV  ILP  is  not  saving  the  DoD  money,  and  the 
Operational  Availability  has  not  increased. 

The  Operational  Availability  was  studied  in  a  manner  similar  to  that  of  the 
CPUU.  We  found  that  over  the  period  studied,  the  Operational  Availability 
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showed  a  very  slight  downward  trend.  However,  the  decrease  was  insignificant 
and  Operational  Availability  never  dropped  below  90%  during  the  time-period 
studied. 

In  conducting  these  analyses,  the  major  finding  was  that  the  data  was 
inconsistent  and  missing  numerous  values.  Specifically,  the  usage  data,  which 
was  vitally  important  to  studying  the  CPUU,  was  missing  monthly  entries  and  had 
numerous  obvious  errors,  such  as  a  monthly  reading  being  lower  than  the 
previous  monthly  reading.  Problems  like  this  caused  difficulty  and  lots  of  data 
had  to  be  eliminated  or  corrected  in  order  to  conduct  the  analysis. 

D.  ORGANIZATION  OF  THIS  THESIS 

This  thesis  describes  the  analytical  methodology,  findings,  and 
recommendations  for  achieving  long-term  product  support  for  the  Army  HMMWV. 
Chapter  II  of  this  thesis  provides  a  brief  synopsis  of  the  different  reports  and 
analyses  that  were  read  while  conducting  this  study.  Chapter  III  provides  an  in- 
depth  discussion  of  the  different  data  sources  used  and,  more  importantly,  the 
problems  encountered  with  the  data.  The  analysis  approach,  methodology,  and 
resulting  models  are  present  in  Chapter  IV.  Finally,  Chapter  V  closes  the  report 
by  drawing  conclusions  and  making  recommendations  for  a  successful  future  of 
the  HMMWV  and  other  DoD  weapons  systems. 
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II.  LITERATURE  REVIEW 


Since  the  inception  of  Performance-Based  Logistics,  a  number  of  studies 
have  examined  its  performance.  The  Government  Accountability  Office  (GAO) 
has  conducted  two  separate  analyses  of  PBL’s,  one  in  2005  and  one  in  2008. 
(U.S.  Government  Accountability  Office,  2005,  2008)  The  RAND  Corporation 
has  also  done  studies  on  PBL,  as  have  many  other  analysis  organizations, 
including  the  U.S.  Army  Materiel  Systems  Analysis  Activity,  the  Office  of  the 
Under  Secretary  of  Defense,  independent  industry,  and  academia.  (RAND 
Corporation,  2008;  Chapman,  2007;  U.S.  Department  of  Defense,  Deputy  Under 
Secretary  of  Defense  for  Logistics  and  Materiel  Readiness,  2009).  But  few  of 
these  studies  have  examined  the  DoD-prescribed  metrics  for  their  analysis. 

A.  PRODUCT  SUPPORT  ASSESSMENT  TEAM 

The  most  recent  high-level  study  conducted  on  Performance-Based 
Logistics  was  released  by  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  (USD  AT&L)  in  November  of  2009.  This  report  was 
created  by  a  working  group  of  military,  civilian  DoD,  industry,  and  academic 
analysts  designated  as  the  Product  Support  Assessment  Team  (PSAT).  While 
the  report  was  very  broad  and  covered  more  topics  than  just  costs  and 
availability,  one  of  the  major  findings  was:  “Performance-based  product  support 
strategies  consistently  deliver  improved  materiel  readiness,  but  assessing  the 
true  cost  of  both  traditional  and  performance-based  strategies  is  difficult,  if  not 
impossible,  given  current  financial  systems.”  (U.S.  Department  of  Defense, 
Deputy  Under  Secretary  of  Defense  for  Logistics  and  Materiel  Readiness,  2009, 
p.  9)  The  major  issue  the  PSAT  had  with  this  analysis  was  that  the  data  for  the 
cost  analysis  was  either  unavailable  or  inconclusive.  Furthermore,  across  the 
services,  there  was  no  standard  way  of  tracking  costs. 

Another  interesting  finding  from  this  report  was  the  fact  that  acquisition 
and  sustainment  communities  do  not  have  shared  goals  and  are  funded 
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separately.  (U.S.  Department  of  Defense,  Deputy  Under  Secretary  of  Defense 
for  Logistics  and  Materiel  Readiness,  2009,  p.  26)  For  example,  acquisition 
organizations  have  traditionally  purchased  weapons  systems  from  the  lowest 
bidder.  While  this  may  provide  savings  for  the  procurement  portion  of  the  life 
span  of  the  system,  it  may,  in  turn,  incur  more  Operation  and  Support  (O&S) 
costs  later  in  the  life-cycle.  If  the  acquisition  groups  and  logistical  support  groups 
worked  more  cohesively  and  had  common  goals,  life-cycle  cost  savings  could  be 
more  easily  achieved.  In  many  cases,  more  costs  up  front  could  lead  to  large 
savings  in  O&S  costs  over  the  life  span  of  the  system.  PBL  contracts  force  the 
acquisition  and  logistic  organizations  to  work  together  and  come  to  agreement  on 
some  common  goals. 

The  PSAT  was  asked  to  conduct  the  analysis  on  analogous  systems.  In 
other  words,  they  grouped  systems  by  mission  capability,  such  as  land  assault 
(AAV,  Bradley,  Stryker,  Ml  FOV,  and  Ml  Abrams),  land  transport  (FMTV,  MTVR, 
HEMTT,  HMMWV  (Army),  and  HMMWV  (USMC)),  etc.  However,  comparing 
different  systems,  produced  by  different  companies  and  at  different  stages  in 
their  life  cycles,  is  problematic.  Ideally,  a  single  system  could  be  examined  both 
with  PBL  in  place  and  without.  For  example,  a  lower  cost  per  unit  usage  for  the 
HMMWV  than  for  the  MTVR  might  suggest  that  the  former’s  maintenance  plan  is 
more  efficient.  However,  the  MTVR  might  be  inherently  more  expensive  to 
maintain,  perhaps  because  parts  cost  more  or  because  the  vehicles  are  used 
differently.  The  PSAT  used  two  primary  indexes  for  grading  the  systems: 
Sustained  Readiness  Improvement  (SRI)  and  Sustained  Cost  Management 
(SCM).  The  SRI  is  a  binary  metric  that  was  calculated  by  taking  the  average 
availability  change  of  all  systems  within  a  category  as  the  baseline.  An 
individual  system  earned  a  value  of  one  for  any  year  in  two  distinct  ways.  First,  a 
system  scored  a  one  if  it  experienced  an  increase  in  availability  from  one  year  to 
the  next.  Second,  a  one  was  also  scored  if  the  system  decreased  in  availability 
at  a  rate  less  than  the  average  availability  change  for  that  category.  Otherwise,  a 
zero  was  scored  for  that  year.  The  average  of  the  ones  and  zeros  over  the 
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period  study  produced  the  SRI.  (U.S.  Department  of  Defense,  Deputy  Under 
Secretary  of  Defense  for  Logistics  and  Materiel  Readiness,  2009,  p.  83)  The 
SCM  was  calculated  using  a  similar  method  except  that  Cost  Per  Unit  Usage 
(CPUU)  was  used  for  each  system  and  in  order  to  earn  a  one,  the  system’s  cost 
had  to  decrease  from  year  to  year  or  increase  at  a  rate  smaller  than  the  average 
CPUU  change.  An  example  of  why  this  could  be  a  problem  is  shown  in  Table  1 . 

The  PSAT  found  numerous  flaws  within  DoD  that  needed  to  be  corrected 
and  made  several  suggestions  on  how  improving  this  process  could  be 
implemented.  It  also  pointed  out  that  this  would  require  strong  leadership,  open- 
mindedness,  a  reform-driven  DoD-Congressional  partnership,  and  a  collaborative 
DoD-Industry  relationship.  (U.S.  Department  of  Defense,  Deputy  Under 
Secretary  of  Defense  for  Logistics  and  Materiel  Readiness,  2009,  p.  12) 

Table  1  shows  how  the  SRI  used  by  PSAT  could  give  a  false  indication 
that  two  systems  are  performing  at  a  similar  level.  Consider  two  systems  with 
Availability,  as  shown  in  Table  1 . 
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System  1: 

Year  1 

Year  2 

Year  3 

Year  4 

Year  5 

Year  6 

Year  7 

Average 

Readiness  improvement 

Availability 

80% 

79% 

78% 

79% 

78% 

77% 

78% 

78% 

+  4 

%  Change 

-1% 

-1% 

+1% 

-1% 

-1% 

+1% 

Score 

0 

1 

1 

0 

1 

1 

2 

67% 

System  2: 

Year  1 

Year  2 

Year  3 

Year  4 

Year  5 

Year  6 

Year  7 

Average 

Readiness  improvement 

Availability 

74% 

75% 

60% 

63% 

65% 

64% 

65% 

67% 

+  4 

%  Change 

+1% 

-20% 

+5% 

+3% 

-2% 

+2% 

Score 

1 

0 

1 

1 

0 

1 

2 

67% 

Average 

77% 

77% 

69% 

71% 

72% 

71% 

72% 

%  Change 

0% 

-11% 

+3% 

+1% 

-1% 

+1% 

Table  1:  SRI  for  Two  Different  Weapons  Systems 
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So,  according  to  the  PSAT  results,  both  systems  receive  a  grade  of  67%; 
however,  clearly  system  #1,  which  has  an  overall  average  of  78%  availability,  is 
the  better  system.  A  graph  of  the  corresponding  Sustained  Readiness 
Improvement  and  the  Operational  Availability  are  shown  in  Figure  1  and  Figure  2. 


Sustained  Readiness  Improvement 


•System  1: 
•System  2: 
•Index 


Figure  1 :  Sustained  Readiness  Improvement  versus  Time  for  Two 
Weapons  Systems  and  Their  Baseline  Index 


System  Availability 


System  1: 
System  2: 


Figure  2:  Operational  Availability  for  Two  Weapons  Systems  Over  Time 
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B.  GOVERNMENT  ACCOUNTABILITY  OFFICER 


The  GAO  has  released  two  reports  regarding  Performance-Based 
Logistics.  (U.S.  Government  Accountability  Office,  2005,  2008)  These  reports 
reach  similar  conclusions.  The  GAO  found  that  while  DoD  guidance  states  that 
program  offices  should  use  business  case  analysis  (BCA)  to  guide  decision 
making  regarding  PBL  contracts,  the  services  are  not  consistent  and  in  many 
cases  simply  did  not  conduct  a  BCA. 

In  the  2005  study,  only  one  of  the  fifteen  program  offices  reviewed  actually 
updated  the  BCA  in  accordance  with  DoD  instruction.  One  of  the  two  major 
recommendations  in  this  report  to  the  DoD  was  “to  reaffirm  DoD  guidance  that 
program  offices  update  the  business  case  analysis  following  implementation  of  a 
performance-based  logistics  arrangement  and  develop  procedures,  in 
conjunction  with  the  military  services,  to  track  where  program  offices  that  enter 
into  those  arrangements  validate  their  business  case  decisions  consistent  with 
DoD  guidance.”  (U.S.  Government  Accountability  Office,  2005,  p.  12)  Three 
years  later,  of  the  29  PBL  arrangements  the  GAO  studied,  about  half  of  them 
either  did  not  use  a  business  case  analysis  or  were  unable  to  provide 
documentation  of  one.  (U.S.  Government  Accountability  Office,  2008,  p.  5) 
Clearly,  the  problem  with  business  case  analysis  has  not  been  fixed.  It  may  be 
that  the  guidance  for  these  business  case  analyses  needs  to  be  revised  or 
perhaps  the  program  offices  are  not  appropriately  equipped  to  conduct  them. 
This  is  an  area  that  still  needs  to  be  addressed  and  corrected  in  order  for  studies 
such  as  this  to  make  any  substantial  conclusions  regarding  the  effectiveness  of 
PBL  contracts. 

The  GAO  also  found  in  both  reports  that  the  data  used  to  evaluate  these 
contracts  is  not  sufficient  to  draw  conclusions.  A  common  belief  identified  in 
these  reports  was  that  a  fixed  cost  contract  would  always  be  cheaper  than  a 
traditional  contract  and  therefore  the  program  offices  did  not  bother  tracking 
costs.  The  program  offices  also  acknowledged  limitations  in  their  own 
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information  systems  in  providing  reliable  data  to  monitor  contractor  cost  and 
performance.  Furthermore,  they  stated  that  they  had  more  confidence  in  the 
contractors’  systems  than  in  their  own  regarding  accuracy  and  completeness. 
(U.S.  Government  Accountability  Office,  2005,  pp.  9-10)  According  to  the  2008 
report,  this  same  problem  persisted.  It  stated,  “Program  managers  often  lacked 
detailed  and  standardized  cost  data.”  (U.S.  Government  Accountability  Office, 
2008,  p.  32)  We  encountered  this  same  lack  of  data  problem  as  well,  and  an 
entire  chapter  of  this  study  has  been  dedicated  to  the  different  problems 
encountered.  Ultimately,  the  DoD  has  to  lay  down  and  enforce  the  standards 
established  in  order  to  track  this  data  accurately  and  completely. 

C.  RAND  CORPORATION 

In  2008,  the  RAND  Corporation  released  a  study  conducted  on  the  Army 
HMMWV  recapitalization  (RECAP)  program.  As  with  the  current  study,  the 
RAND  study  was  performed  at  the  vehicle  level.  The  key  objectives  of  the  study 
were  to  assess  the  effects  of  age  on  costs  and  availability,  develop  a  tool  that 
determines  optimal  RECAP  times,  and  demonstrate  how  the  tool  can  be  used  to 
produce  recommendations.  (RAND  Corporation,  2008)  While  RAND’s  analysts 
were  not  as  interested  in  cost  per  unit  usage  as  this  study  is,  they  did  determine 
that  the  optimal  replacement  point  for  the  M998  occurred  at  age  12,  yielding  an 
average  cost  per  mile  of  $5.53  over  the  lifetime  of  the  vehicle.  (RAND 
Corporation,  2008,  p.  xiii) 

As  with  all  the  other  studies,  RAND  found  numerous  problems  with  the 
data.  Since  their  study  was  based  on  age,  the  Year  of  Manufacture  (YOM)  was 
important  data.  About  5%  of  the  data  they  collected  did  not  have  a  YOM,  or  it 
contained  a  YOM  that  was  obviously  incorrect.  The  usage  data  was  even  worse. 
In  order  to  put  some  realistic  maximum  limitations  on  this,  they  capped  the  usage 
at  3,000  miles  per  month.  With  this  restriction,  over  24%  of  the  data  had  missing 
or  invalid  usage  elements.  RAND  used  simple  imputation  to  replace  these 
missing  values.  Another  issue  RAND  faced  lay  in  the  part  order  data.  Their  data 
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contained  part  orders  that  were  not  true  HMMWV  parts,  and  therefore  those  were 
excluded.  (RAND  Corporation,  2008,  p.  57)  Fortunately,  RAND’s  data  set  was 
large  enough  that  even  with  these  problems,  they  were  able  to  conduct  a 
thorough  study  and  develop  a  model  for  determining  the  optimal  age  for  RECAP. 

D.  ARMY  MATERIEL  SYSTEMS  ANALYSIS  ACTIVITY  (AMSAA) 

The  Army  Materiel  Systems  Analysis  Activity  (AMSAA)  conducted  a  study 
on  the  HMMWV  ILP  and  released  a  report  in  October  of  2007.  This  study  was 
conducted  for  Headquarters  Army  Materiel  Command  and  included  five  different 
performance  metrics.  These  performance  metrics  were:  Stockout  Rate, 
Turnaround  Time,  Code  G  Vehicle  Rate,  Customer  Wait  Time,  and  Quality 
Defect  Rate.  There  were  also  four  cost  metrics:  Repair  Parts  Cost,  Supply  Chain 
Operating  Cost,  Investment/Other  Additional  Cost,  and  Inventory  Holding  Cost. 
According  to  this  study,  the  cost  savings  to  the  government  from  the  inception  of 
the  ILP  in  January  of  2006  to  December  of  2006  was  approximately  $800,000. 
(Chapman,  2007,  p.  33).  The  performance  of  the  program  conveyed  consistent 
improvement  according  to  AMSAA.  The  major  recommendations  that  AMSAA 
had  upon  conclusion  of  their  study  all  centered  around  data  collection.  They 
found  that  the  data  for  the  baseline  assessment  was  not  available,  that  adequate 
time  to  conduct  the  baseline  data  collection  was  not  permitted  prior  to  the 
performance  based  agreement  (PBA)  being  negotiated,  and  that  baseline  metrics 
needed  to  be  established  prior  to  the  PBA  being  written.  As  with  all  other  studies 
that  were  reviewed,  the  lack  of  data  or  inaccuracy  of  data  was  a  major  hurdle  to 
overcome. 

E.  DEFENSE  ACQUISITION  UNIVERSITY 

In  order  to  gain  insight  into  Performance-Based  Logistics,  the  “PBL  Guide” 
provided  by  Defense  Acquisition  University  was  reviewed  along  with  the 
governing  DoD  instruction,  5000  series.  According  to  DoD  Instruction  5000.01, 
program  managers  “shall  develop  and  implement  performance-based  logistics 
strategies  that  optimize  total  system  availability  while  minimizing  costs  and 
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logistics  footprint.”  This  is  interesting  to  note,  considering  that,  as  previously 
stated,  only  20%  of  current  weapons  systems  are  under  a  PBL  contract. 
According  to  the  “PBL  Guide”  and  many  other  reports,  the  preferred  method  of 
product  support  strategy  is  performance-based  logistics.  (Defense  Acquisition 
University,  2005,  p.  vii) 

The  “PBL  Guide”  offers  information  for  program  managers  to  use  as  they 
shift  to  this  new  strategy.  It  offers  guidance  on  developing  a  team,  determining 
performance  outcomes,  supply  chain  management  strategy,  and  awarding 
contracts,  along  with  many  other  topics.  Additionally,  this  guide  provides 
information  on  the  five  primary  metrics  for  PBL  contracts:  1)  Operational 
Availability,  2)  Operational  Reliability,  3)  Cost  per  Unit  Usage,  4)  Logistics 
Footprint,  and  5)  Logistics  Response  Time.  Also  presented  in  this  guide  book 
are  several  success  stories  of  PBL  contracts  and  a  section  on  best  practices 
determined  from  these  success  stories. 

F.  CONCLUSION 

In  conclusion,  many  different  studies  were  reviewed  and  one  resounding 
conclusion  consistently  appeared.  The  data  that  was  available  to  conduct  cost 
and  performance  analysis  is  limited  and  of  dubious  quality.  Because  of  this,  the 
ability  of  the  different  teams  to  provide  substantial  and  conclusive  results  was 
very  limited.  Furthermore,  they  were  unable  to  study  the  DoD  prescribed  metrics 
for  PBL  contracts.  In  the  case  of  the  PSAT  study,  the  scope  of  the  study  was 
extremely  large  and  made  an  attempt  to  compare  analogous  systems. 
Unfortunately,  while  these  systems  are  similar,  they  have  their  differences  as 
well,  and  therefore  it  is  hard  to  make  exact  comparisons.  Ultimately,  to 
determine  if  PBL  contracts  are  working,  analysts  will  have  to  study  systems  that 
have  used  both  methods  of  life-cycle  support.  This  allows  for  a  fair  comparison 
amongst  the  two  methods.  Also,  the  metrics  prescribed  by  DoD  should  be  the 
focus  of  these  future  studies.  The  bottom  line  is  that  in  order  to  conduct  a 
thorough  and  significant  analysis,  there  must  be  accurate  and  complete  data 
available. 
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III.  DATA  ISSUES 


A  major  difficulty  in  this  study  in  assessing  the  metrics  for  the  HMMWV  is 
the  inconsistencies  found  in  the  data.  There  are  many  obvious  errors  that  were 
found  within  the  data  sets.  This  problem  was  discussed  in  Chapter  II.  The  data 
collection  process  requires  drastic  improvement.  Studies  such  as  this  are  only 
as  good  as  the  data  they  can  use.  This  chapter  discusses  the  problems  in  the 
data  that  was  obtained  on  the  HMMWV  for  both  the  U.S.  Army  and  U.S.  Marine 
Corps. 

A.  UNITED  STATES  ARMY 

The  metrics  studied  for  the  HMMWV  were  Operational  Availability  and 
Cost  Per  Unit  Usage  (CPUU).  These  metrics  were  explored  on  the  M998  and 
Ml 097  models,  which  are  the  two  models  for  which  the  Army  ILP  was  originally 
written.  The  data  was  collected  from  Operating  and  Support  Management 
Information  System  (OSMIS)  and  the  Logistics  Integrated  Database  (LIDB).  The 
data  collected  from  OSMIS  provided  a  summary  of  quarterly  costs  for  each  Major 
Army  Command  (MACOM)  and  we  were  able  to  capture  data  from  1993  to 
present.  LIDB  provided  a  detailed  look  at  costs  broken  down  by  individual  Work 
Order  Numbers  (WONs).  LIDB  also  provided  usage  data  for  each  vehicle  by 
serial  number.  The  Operational  Availability  data  in  LIDB  is  reported  by  each  Unit 
Identification  Code  (UIC)  that  maintains  that  particular  style  of  vehicle  and  is 
recorded  monthly.  All  data  collected  from  LIDB  included  the  years  2000  to  2009. 
However,  due  to  minimal  data  recorded  between  2000  to  2003,  only  2004  to 
2009  were  analyzed. 

1.  Operating  and  Support  Management  Information  System 
(OSMIS) 

The  data  collected  from  OSMIS  had  numerous  anomalies  that  had  to  be 
studied  closely  to  determine  why  they  occurred  or  whether  the  data  was  simply 
incorrect.  For  instance,  10%  of  the  line  items  in  the  OSMIS  data  contained  zeros 
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in  the  Activity  (amount  of  usage)  field.  This  typically  would  cause  the  Cost  Per 
Unit  Usage,  which  is  the  sum  of  all  the  costs  for  a  vehicle  divided  by  the  miles 
traveled,  to  be  infinite.  However,  these  values  were  recorded  as  zero.  This 
clearly  is  an  error  because  there  were  reported  costs  for  those  years  and 
therefore  the  vehicles  had  presumably  been  used.  It  should  be  noted  that  most 
of  these  zeros  fell  in  the  time  frame  from  1993-1998.  Furthermore,  the 
maximum  CPUU  during  the  time  frame  studied  was  $16.4  billion.  Approximately 
10%  of  the  reported  CPUU  values  were  over  $10.00  per  mile.  Such  a  vehicle 
operated  for  1200  miles  per  year  would  cost  around  $12,000  dollars  per  year; 
over  the  course  of  a  15-year  life  span,  O&S  costs  would  be  around  $180,000 
dollars  at  that  rate.  That  does  not  include  the  purchase  price.  From  these 
observations  of  the  CPUU  reported  in  OSMIS,  we  decided  that  this  10%  of  the 
data  was  defective  and  therefore  needed  to  be  removed.  A  plot  of  average 
CPUU  for  Army  HMMWV  is  shown  in  Figure  3. 
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Army  HMMWV  Cost  Per  Unit  Usage 
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The  CPUU  for  the  M998  is  much  more  consistent  than  that  of  the  Ml 097 
and  this  can  probably  be  attributed  to  the  fact  that  there  are  substantially  more 
M998  vehicles  than  Ml 097  vehicles  for  most  of  the  years  studied.  One  last 
interesting  note  for  the  CPUU  is  the  drastic  spike  in  2009  for  the  M998  model  and 
in  1999  and  2001  for  the  Ml  097  model. 

Another  example  of  the  inconsistencies  lay  in  the  number  of  HMMWV’s 
reported  in  OSMIS  that  are  displayed  in  Figure  4.  This  type  of  study  requires  a 
large  number  of  vehicles  in  order  to  ensure  accurate  results.  In  the  final  data 
sets  used  for  the  study,  there  were  ample  vehicles,  but  the  point  of  this 
paragraph  is  to  show  inconsistencies  in  the  data.  Note,  the  drastic  change  in  the 
reported  number  of  M998  models  between  Quarter  4  of  1998  and  Quarter  1  of 
1999.  This  drop  was  over  30,000  vehicles.  While  there  is  no  clear  explanation 
for  this,  it  could  possibly  be  attributed  to  some  vehicles  being  re-designated  to  a 
different  model  number  due  to  an  upgrade  or  addition  to  the  vehicle.  Numerous 
calls  and  emails  yielded  no  other  possible  conclusions  from  the  Program 
Manager  and  AMC.  For  example,  if  a  new  communication  system  is  installed  on 
a  vehicle,  it  is  re-designated  from  Ml 097  to  M1097A1.  Also  notable  in  this  graph 
is  the  spike  in  numbers  for  the  Ml 097  during  1998  followed  by  a  drop  in  1999. 
While  the  drop  is  not  nearly  significant  as  that  for  M998  in  1999,  there  is  another 
substantial  drop  between  Quarters  1  and  2  in  2006  for  the  M998  model.  This  is 
when  the  ILP  first  began.  One  trend  that  is  clear  from  this  graph  is  the  rise  in 
numbers  of  Ml 097  models  and  decrease  of  M998  models  beginning  in  2006. 
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Numberof  M1097 


The  final  area  of  interest  from  the  OSMIS  data  was  the  Cost  Per 
System/Unit  (CPU).  CPU  is  the  total  O&S  costs  for  an  individual  system.  While 
this  was  not  one  of  the  primary  metrics  studied,  the  data  was  observed  initially 
just  to  see  how  it  coincided  with  the  Cost  Per  Unit  Usage.  It  seems  that  these 
two  metrics  should  have  some  similarities  especially  over  such  a  large  sample 
size.  The  graph  of  the  CPU  is  displayed  in  Figure  5.  While  there  appears  to  be  a 
rising  trend  across  the  entire  time  frame,  it  is  also  easy  to  see  the  large  spikes 
from  quarter  to  quarter.  Furthermore  the  increases  and  decreases  do  not 
coincide  with  the  increases  and  decreases  in  the  CPUU  graph  in  Figure  3. 
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Army  HMMWV  Cost  Per  Unit 


Figure  5:  Army  HMMWV  Cost  Per  Unit  for  M998  and  M1097  from  1996  to  2009 
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While  there  appears  to  be  no  consistency  in  this  data,  the  two  models  do 
seem  to  follow  similar  trends.  In  other  words,  CPU  for  the  M998  and  Ml 097 
models  appear  to  be  rising  and  falling  at  the  same  time.  From  2000  to  2007,  the 
range  of  CPU  for  M998  was  $779.52  to  $1528.49.  This  is  a  significant  range, 
particularly  considering  that  all  these  values  have  been  adjusted  to  FY  2011 
dollars. 

2.  Logistics  integrated  Database  (LIDB) 

The  Logistics  Integrated  Database  contained  vehicle  level  data  that  was 
much  more  useful.  Data  on  both  cost  and  Operational  Availability  was  obtained 
for  the  years  2000-2009.  Although  this  was  by  far  the  best  data  we  were  able  to 
obtain,  it  still  had  some  significant  problems.  First,  the  cost  data  for  these  years 
had  268,348  Work  Order  Numbers  with  294  of  those  being  duplicates.  However, 
the  categorical  data  had  only  264,492  WONs  with  59  duplicates.  Costs 
associated  with  a  WON  that  did  not  exist  in  the  categorical  data  raised  some 
concern.  Without  a  WON  in  the  categorical  data,  it  was  impossible  to  determine 
which  serial  number  those  costs  could  be  associated  with. 

The  next  major  problem  with  this  data  was  the  fact  that  the  first  four  years 
of  the  data  contained  very  few  WONs.  That  left  us  with  only  two-years’  worth  of 
data  to  establish  the  baseline  before  the  ILP  was  implemented.  A  histogram  of 
the  number  of  WONs  by  year  is  shown  in  Figure  6.  In  2000,  there  were  two 
WONs  in  the  data;  in  2001  only  one;  and  there  were  sixteen  in  2002.  These 
numbers  are  not  visible  in  the  graph  due  to  the  scale.  The  important  conclusion 
is  that  prior  to  2004,  WONs  were  not  being  tracked  or  entered,  and  therefore, 
these  years  cannot  be  used  for  any  significant  analysis. 
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Histogram  of  WONs  by  Year 


2000  2002  2004  2006  2008 


Figure  6:  Histogram  of  Work  Order  Numbers  for  2000-2009 

Finally,  of  the  over  250,000  work  orders  associated  with  costs,  over 
12,000  of  them  had  a  zero  in  the  total  work  order  cost  field,  and  over  26,000  of 
them  were  missing  the  serial  number  for  the  vehicle.  Slightly  fewer  than  140,000 
of  them  had  valid  six-digit  serial  numbers  (before  we  implemented  correction 
algorithms).  Ultimately,  this  was  the  best  we  were  able  to  collect  and  after 
corrections  to  serial  numbers,  that  is  detailed  in  Appendix  A,  we  used  just  under 
180,000  WONs  for  this  study. 
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B.  UNITED  STATES  MARINE  CORPS 


1.  Visibility  and  Management  of  Operating  and  Support  Costs 
(VAMOSC) 

The  metrics  studied  for  the  USMC  HMMWV  were  the  same  as  with  the 
Army:  Operational  Availability  and  Cost  Per  Unit  Usage.  The  Marine  Corps  data 
was  gathered  from  the  Navy  Visibility  And  Management  of  Operating  and 
Support  Cost  (VAMOSC)  database  at  the  Naval  Center  for  Cost  Analysis 
(NCCA).  Data  was  gathered  on  eight  different  models  of  the  HMMWV.  The 
years  from  1998  to  2008  were  available  for  download;  however,  some  of  the 
models  did  not  appear  in  all  of  these  years.  The  data  obtained  was  summary 
statistics  by  model  and  year.  There  were  88  different  combinations  of  model  and 
year  obtained.  As  with  the  Army  data,  several  anomalies  were  noted  upon  initial 
collection  of  the  data. 

The  most  significant  problem  with  this  data  set  was  the  fact  that  the 
“OPTEMPO”  or  usage  was  reported  for  both  deployed  and  regular  units; 
however,  the  inventory  was  only  reported  for  the  entire  fleet  of  that  particular 
model.  This  makes  it  difficult  to  determine  the  average  usage  per  vehicle,  which 
is  needed  to  determine  the  cost  per  unit  usage.  Furthermore  the  units  for  the 
OPTEMPO  were  inconsistent  and  in  many  cases  not  documented.  Of  the  88 
organizations  listed  in  line  items  for  regular  (non-deployed)  vehicles,  77  of  them 
were  reported  in  miles  per  year,  five  were  reported  in  hours  per  year,  and  six 
were  left  blank.  For  the  deployed  units,  35  were  reported  in  miles  per  year,  one 
was  reported  in  hours  per  year,  and  52  were  left  blank.  The  52  that  did  not  report 
could  be  attributed  to  that  particular  vehicle  style  not  having  any  deployed 
vehicles  for  that  particular  year.  Based  on  these  numbers  and  various 
conversations  with  Marines  who  have  dealt  with  HMMWV’s  in  the  past,  it  is 
believed  that  the  correct  unit  is  in  miles  per  year. 

As  with  the  Army,  the  Marines’  inventories  or  counts  were  another 
significant  problem  area.  Of  the  88  line  items  gathered  from  VAMOSC,  only  73% 
included  an  inventory;  however,  all  88  included  at  least  one  of  the  five  different 
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cost  fields  included  in  the  data.  A  graph  of  the  inventories  is  included  in  Figure  7, 
and  while  the  drastic  drops  seen  in  the  Army  data  are  not  as  evident  here,  there 
are  several  oddities  that  raise  concerns.  For  instance:  for  the  D1125  model  for 
years  2005  to  2007,  the  inventory  varied  from  669  to  1175  to  854.  A  similar 
problem  occurred  for  the  D0187  model  during  the  same  years;  the  inventory 
varied  from  17  to  357  to  74.  For  FY-2009,  no  inventories  were  reported  for  any 
model.  In  summary,  the  USMC  data  was  not  analyzed  for  this  thesis  due  to  the 
lack  of  vehicle  level  data. 


25 


USMCHMMWV  Inventory 


?. son 
2000 
1500 
1000 
500 
0 


1998  1999  2000  2031  2002  2003  2004  2005  2006  2007  2008  2009 


•DOC'Ol 

■D0187 

•D0188 

■D1001 

D1C02 

•D1125 

D1159 


Figure  7:  USMC  HMMWV  Inventory  Levels  for  1998  to  2008 
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The  final  observation  from  this  data  was  the  Cost  Per  Unit  (CPU).  The 
most  interesting  observation  in  this  area  was  with  the  D0187  model.  Over  the 
course  of  6  years,  the  CPU  data  varied  drastically  and  was  more  than  5-7  times 
higher  than  the  other  models’  average  in  4  out  of  6  years.  A  graph  of  the  CPU  is 
shown  Figure  8,  and  the  anomaly  that  occurred  with  D0187  is  clear  to  see. 
There  is  also  a  trend  beginning  in  2005  that  shows  an  increase  in  CPU. 
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USMC  HMMWV  Cost  Per  Unit 
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Figure  8:  USMC  HMMWV  Cost  Per  Unit  for  1998  to  2008 
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C.  SUMMARY 

Data  obtained  from  the  Army  and  Marine  Corps  was  full  of  anomalies. 
From  unexplained  drops  in  inventory  to  differences  in  Cost  Per  Unit  of  over 
$25,000,  the  data  contained  many  inconsistencies.  While  there  is  plenty  of  data 
available  on  weapons  systems  from  the  various  databases  across  the  services,  it 
appears  there  needs  to  be  some  quality  control  implemented  at  the  entry  level. 
For  maintenance  issues,  jobs  are  typically  entered  into  the  system  by  personnel 
of  rank  E-5  and  below  and  then  reviewed  by  Non-Commissioned  Officers  and 
again  by  Officers.  Unfortunately,  this  review  process  is  not  correcting  the 
multitude  of  errors  and  needs  to  be  revised  across  the  services.  Perhaps  with 
new  technology  available,  the  multiple  entry  points  could  have  drop-down  menus 
and  the  ability  to  warn  the  user  of  a  faulty  input.  Ultimately,  for  DoD  to  conduct 
thorough  and,  more  importantly,  conclusive  studies,  we  recommend  that  the 
issues  addressed  above  regarding  the  data  be  corrected. 
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IV.  ANALYSIS 


The  analysis  focused  on  two  of  the  six  metrics  used  to  measure  PBL’s: 
Cost  Per  Unit  Usage  (CPUU)  and  Operational  Availability.  We  examined  the 
Cost  Per  Unit  Usage  over  time  to  determine  if  any  cost  savings  could  be 
attributed  to  the  ILP.  Operational  Availability  was  studied  to  find  any  trends 
before  and  after  the  inception  of  the  partnership  with  AM  General.  All  the 
analysis  was  conducted  in  R  and  Excel  2007  using  the  data  gathered  from  the 
Army  LIDB.  Due  to  the  large  number  of  missing  data  points,  some  of  the  data 
had  to  be  repaired  and  those  repairs  are  documented  in  Appendix  A. 

Two  different  models  from  the  Army  were  studied,  the  Ml 097  and  M998. 
Over  45,000  vehicles  were  used  for  this  study,  33,761  M998s  and  14,832 
Ml 097s.  The  data  covered  the  years  2004  to  2009.  After  repairs  and  deletions, 
we  had  nearly  128,000  work  orders  for  M998’s  and  41,000  work  orders  for  the 
M1097’s. 

A.  COST  PER  UNIT  USAGE 

The  cost  per  unit  usage  was  difficult  to  study  because  the  usage  data  was 
missing  multiple  values  and  had  countless  anomalies.  After  repairs  to  the  data, 
there  was  still  variability.  The  mean  usage  per  year  per  was  914  miles  for  Ml  097 
and  1,242  for  the  M998.  A  table  of  the  number  of  vehicles  having  a  certain  level 
of  usage  for  a  year  is  shown  in  Table  2.  There  were  over  28,000  times  when  a 
vehicle  recorded  zero  usage  for  the  year.  The  average  vehicle  usage  decreased 
over  the  period  studied.  A  plot  of  both  models’  average  yearly  usage  is  shown  in 
Figure  9. 


Miles  of 
Usage 

0 

1-1,000 

1,001- 

5,000 

5,001- 

10,000 

>10,000 

M998 

20,547 

26,640 

21,362 

2,515 

851 

M1097 

8,001 

11,999 

5,673 

570 

254 

Table  2:  Number  of  Vehicles  with  Established  Mileage  Per  Year 
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Mean  Usage  by  Year 


Figure  9:  Mean  Usage  per  Year 


A  boxplot  of  the  log  of  the  usage  is  provided  in  Figure  10  and  Figure  1 1  for 
the  Ml 097  and  M998,  respectively.  We  chose  the  log  of  the  usage  value 
because  the  spread  of  the  values  was  so  large  it  would  have  been  difficult  to 
display  otherwise.  Note  that  in  these  plots  values  of  zero  were  excluded.  This 
eliminated  from  between  9%  to  42%  of  the  data  for  any  given  year.  The 
percentage  of  zero  values  is  shown  below  the  year  in  parentheses.  Notice  that 
the  percentages  increase  over  time.  These  plots  indicate  a  downward  drift  in  the 
usage  of  the  period  studied. 
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Log  of  Usage 


Boxplot  of  Log  of  M998  Usage 


(15%)  (18%)  (27%)  (38%)  (40%)  (42%) 


Figure  10:  Boxplot  of  Log  of  Usage  for  M998  excluding  all  values  of 

zero  from  original  data 
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Log  of  Usage 


Boxplot  of  Log  of  M1097  Usage 


(9%)  (14%)  (32%)  (26%)  (31%)  (34%) 


Figure  1 1 :  Boxplot  of  Log  of  Usage  for  M1097  excluding  all  values 

of  zero  from  original  data 

Despite  the  problems  with  the  usage  data,  we  examined  the  CPUU  for 
both  vehicle  models.  The  goal  was  to  use  the  years  prior  to  the  implementation 
of  the  ILP  as  a  baseline  and  then  observe  how  the  metric  changed  with  respect 
to  the  baseline  after  implementation. 
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All  CPUU  values  discussed  in  this  chapter  account  for  inflation  and  are 
displayed  in  Fiscal  Year  2011  dollars.  The  adjustments  were  made  in 
accordance  with  the  Naval  Center  for  Cost  Analysis  using  the  Operations  and 
Maintenance  Army  Appropriations  index.  (Naval  Center  for  Cost  Analysis,  2010) 

1.  Summary  Statistics 

Table  3  summarizes  the  CPUU  before  and  after  the  ILP  was  implemented. 
The  year  2006  is  excluded  in  column  three  to  eliminate  the  possibility  of  the 
CPUU  being  skewed  in  that  year  due  to  initial  start-up  costs  for  the  program. 
However,  notice  that  when  2006  is  included,  the  CPUU  was  actually  lower.  We 
conjecture  that  when  the  program  initially  started,  there  were  still  parts  on  the 
shelves  at  the  depots  that  had  been  procured  through  the  old  system,  and  the 
technicians  were  using  these  despite  the  fact  that  AM  General  was  under 
contract  to  provide  the  parts.  The  key  takeaway  is  that  the  CPUU  in  2006  cannot 
be  accurately  represented  and  should  be  viewed  cautiously  when  trying  to  draw 
conclusions  about  the  HP’s  success  (or  lack  thereof). 


2004-2005 

2006-2009 

2007-2009 

M998 

$98.14 

$111.19 

$119.02 

M998  5%  Trim 

$5.72 

$10.44 

$11.63 

Ml  097 

$80.32 

$107.56 

$111.75 

Ml 097  5%  Trim 

$6.21 

$8.39 

$8.81 

Table  3:  CPUU  Before  and  After  ILP  Implementation 


Based  on  these  averages,  the  overall  CPUU  for  both  units  appears  to  be  rising 
over  time.  We  used  the  5%  trimmed  means  because  there  were  a  small 
percentage  of  very  large  values  that  had  significant  influence  on  the  CPUU 
values.  The  distribution  of  the  CPUU  values  is  shown  in  Table  4. 
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Model  & 

Years 

CPUU  FY11$ 

$0.00- 

$5.00 

$5.01- 

$10.00 

$10.01- 

$50.00 

$50.01- 

$100.00 

$100.01- 

$1000.00 

>$1000.00 

NA's 

M998 

Count 

33633 

4864 

6502 

1402 

2224 

850 

20547 

% 

68% 

10% 

13% 

3% 

4% 

2% 

M998 

2004-2005 

Count 

15894 

2006 

2438 

481 

792 

356 

4643 

% 

72% 

9% 

11% 

2% 

4% 

2% 

M998 

2006-2009 

Count 

17739 

2858 

4064 

921 

1432 

494 

15904 

% 

64% 

10% 

15% 

3% 

5% 

2% 

M1097 

Count 

10441 

2033 

3183 

885 

1190 

369 

8001 

% 

58% 

11% 

18% 

5% 

7% 

2% 

M1097 

2004-2005 

Count 

1006 

127 

234 

64 

58 

20 

225 

% 

67% 

8% 

16% 

4% 

4% 

1% 

M1097 

2006-2009 

Count 

9435 

1906 

2949 

821 

1132 

349 

7776 

% 

57% 

11% 

18% 

5% 

7% 

2% 

Table  4:  CPUU  Counts  and  Percentages  for  Cost  Ranges 
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Figure  12  displays  the  5%  trimmed  average  CPUU  for  both  vehicle  models  for 
each  quarter  over  the  period  studied.  Again,  it  is  clear  that  the  CPUU  appears  to 
be  rising  with  time. 


5%  T rimmed  Mean  CPUU  by  Quarter 


Figure  12:  5%  Trimmed  Mean  CPUU  by  Quarter  from  FY2004  to 

FY2009 

Notice  that  there  are  several  large  spikes  in  the  CPUU,  especially  for  the  Ml 097 
vehicle.  In  the  second  quarter  of  fiscal  year  2008,  there  was  a  large  jump  for 
both  models.  This  graph  displays  how  inconsistent  this  metric  is  and  we  believe 
that  is  attributable  to  variability  in  the  usage  data. 

While  the  quarterly  data  was  not  very  conclusive,  we  were  able  to  see  a 
better  trend  when  using  the  yearly  averages.  Figure  13  and  Figure  14  display 
the  yearly  averages  with  a  trim  level  of  2%,  5%,  and  10%.  Notice  that  in  2006 
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the  CPUU  increased  for  both  vehicles  with  the  exception  of  the  2%  trimmed 
mean  for  the  M998  vehicle.  As  previously  mentioned,  this  was  expected  as  a 
response  to  increased  start-up  costs.  The  following  year,  2007,  the  CPUU 
dropped  for  the  Ml 097,  indicating  that  the  program  may  have  been  achieving 
some  cost  savings.  However,  the  following  two  years  were  higher  and  indicate 
that  the  CPUU  was  growing  with  time.  We  need  to  point  out  that  over  time  the 
average  usage  for  these  vehicles  decreased,  as  shown  in  Figure  9.  So,  for 
routine  maintenance  that  is  based  on  time,  not  mileage,  the  CPUU  would  be 
driven  up.  This  could  explain  why  an  increase  in  the  CPUU  is  seen  during  the 
period  of  study. 


M998  Trimmed  Mean  CPUU 


Figure  13:  M998  Trimmed  Mean  CPUU  from  2004-2009 
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Ml 097  Trimmed  Mean  CPUU 


Figure  14:  M1097  Trimmed  Mean  CPUU  from  2004-2009 

Additionally,  we  looked  at  the  boxplot  of  each  year’s  CPUU  values  side  by 
side.  Looking  at  the  raw  data,  it  was  hard  to  gather  any  useful  information 
because  there  was  such  a  large  number  of  outliers  and  the  ranges  of  values 
were  from  $0.00  to  nearly  $300,000.  To  make  this  a  more  readable  plot,  we 
plotted  the  log  of  the  CPUU.  The  plots  for  both  models  are  shown  in  Figure  15 
and  Figure  16.  Again,  notice  the  upward  drift  over  the  period  studied.  While  this 
does  not  indicate  any  huge  shift  in  costs,  it  clearly  shows  that  the  costs  are  rising 
over  time,  and  that  there  is  no  cost  savings  that  can  be  attributed  to  the  ILP.  This 
may  have  happened  regardless  of  the  ILP,  but  there  is  no  way  to  determine  that. 
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M998  Distributions  by  Year 
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Ml 097  Distributions  by  Year 


2004  2005  2006  2007  2008  2009 


Figure  16:  Year-by-Year  boxplot  of  Log(CPUU)  for  Ml 097 


2.  CPUU  by  Year  of  Manufacture 

Next,  we  looked  at  the  CPUU  based  on  the  Year  of  Manufacture  (YOM). 
Prior  to  examining  this,  we  expected  to  see  some  trends  in  the  data  because  as 
a  vehicle  gets  older  we  expect  it  to  need  repair  more  often  than  a  newer  vehicle. 
This  is  not,  however,  what  we  observed  from  the  data.  The  later  model  vehicles 
had  more  variability  and  their  CPUU  was  higher.  This  may  be  attributed  to  newer 
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parts  being  more  expensive,  the  price  of  labor  rising,  or  new  technology.  Plots  of 
the  two  models’  CPUU  based  on  YOM  are  displayed  in  Figure  17  and  Figure  18. 
Notice  that  the  pre-ILP  CPUU  values  are  lower  for  most  years.  Although  these 
are  10%  trimmed  means,  large  spikes  in  the  CPUU  values  are  still  visible.  The 
one  value  that  is  not  displayed  on  the  Ml 097  chart  was  over  $280  per  mile.  The 
values  after  the  implementation  of  the  ILP  have  less  variability  but  are  generally 
higher  than  those  values  prior  to  the  ILP. 


M998  CPUU  by  Year  of  Manufacture 


Year  of  Manufacture 


—♦—<=2005 

—■—>2005 


Figure  17:  M998  CPUU  by  Year  of  Manufacture  before  and 

after  ILP  Implementation 
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M1097  CPUU  by  Year  of  Manufacture 


Year  of  Manufacture 


-»-<=?oos 

—•—>2005 


Figure  18:  M1097  CPUU  by  Year  of  Manufacture  before  and 

after  ILP  Implementation 

3.  CPUU  by  Year  of  MACOM 

Next,  we  looked  at  the  CPUU  for  each  Major  Army  Command  (MACOM). 
Two  of  the  MACOMs,  Forces  Command  (FORSCOM)  and  National  Guard 
Bureau  (NGB),  owned  70%  of  the  vehicles  studied.  The  remaining  30%  of 
vehicles  were  distributed  among  15  other  MACOMs.  Again,  we  attempted  to 
compare  the  CPUU  before  and  after  the  implementation  of  the  ILP.  The  results 
were  the  same  as  in  the  previous  analyses.  The  CPUU  for  2004  and  2005  was 
cheaper  than  for  2006  to  2009.  For  the  FORSCOM  and  NGB  MACOM’s,  the 
CPUU  was  1.5  to  2.25  times  larger  after  the  implementation  of  the  ILP  than  it 
was  before.  Again,  we  used  a  10%  trimmed  mean  to  eliminate  the  influence  of 
some  of  the  unusually  large  values.  A  summary  of  our  findings  is  shown  in  Table 
5.  A  score  of  one  was  assigned  for  those  MACOM’s  in  which  the  pre-ILP  CPUU 
was  lower  than  the  post-ILP.  A  zero  was  scored  for  the  opposite.  The 
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percentages  at  the  bottom  of  the  “Score”  columns  represent  the  proportion  of 
vehicles  that  belonged  to  a  MACOM  in  which  the  costs  went  up.  This  indicates 
that  the  CPUU  has  risen  since  the  inception  of  the  ILP. 
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M998 

M1097 

MACOM 

Sample 

Size 

% 

< 

=2005 

>2005 

Score 

Sample 

Size 

% 

< 

=2005 

>2005 

Score 

EUSA 

1471 

2.0% 

$ 

2.41 

$ 

13.20 

1 

201 

0.8% 

$ 

1.75 

$ 

34.13 

1 

FORSCOM 

30643 

42.6% 

$ 

5.04 

$ 

11.24 

1 

16548 

62.5% 

$ 

8.08 

$ 

11.74 

1 

IMCOM 

446 

0.6% 

$ 

3.65 

$ 

4.06 

1 

672 

2.5% 

$ 

1.50 

$ 

24.06 

1 

MDW 

16 

0.0% 

$ 

9.72 

$ 

1.57 

0 

1 

0.0% 

NA 

$ 

0.02 

NA 

NETCOM 

417 

0.6% 

$ 

4.23 

$ 

10.61 

1 

188 

0.7% 

$ 

12.21 

$ 

36.85 

1 

NGB 

15227 

21.2% 

$ 

1.85 

$ 

2.73 

1 

2778 

10.5% 

$ 

1.16 

$ 

2.15 

1 

TRADOC 

3281 

4.6% 

$ 

1.02 

$ 

2.44 

1 

327 

1.2% 

$ 

2.26 

$ 

2.69 

1 

USACIDC 

34 

0.0% 

$ 

42.88 

$  280.27 

1 

0 

0.0% 

NA 

NA 

NA 

USAIMA 

34 

0.0% 

$ 

10.97 

$ 

9.23 

0 

0 

0.0% 

NA 

NA 

NA 

USAMC 

53 

0.1% 

$ 

2.81 

$ 

36.31 

1 

1414 

5.3% 

NA 

$ 

29.36 

NA 

USARC 

11943 

16.6% 

$ 

3.74 

$ 

4.66 

1 

2251 

8.5% 

$ 

4.62 

$ 

3.87 

0 

USARCENT 

179 

0.2% 

$ 

2.41 

$ 

22.64 

1 

0 

0.0% 

NA 

NA 

NA 

USAREUR  7 A 

4917 

6.8% 

$ 

2.67 

$ 

5.63 

1 

440 

1.7% 

$ 

4.42 

$ 

5.65 

1 

USARPAC 

2286 

3.2% 

$ 

5.43 

$ 

7.83 

1 

1117 

4.2% 

$ 

2.05 

$ 

23.09 

1 

USARSO  6A 

114 

0.2% 

$ 

0.01 

$ 

0.99 

1 

0 

0.0% 

NA 

NA 

NA 

USASOC 

534 

0.7% 

$ 

2.64 

$ 

7.15 

1 

541 

2.0% 

$ 

12.03 

$ 

5.16 

0 

USINSCOM 

295 

0.4% 

$ 

10.71 

$ 

9.86 

0 

19 

0.1% 

$ 

5.37 

$ 

6.23 

1 

TOTAL 

71890 

99.52% 

26497 

84% 

Table  5:  CPUU  Comparison  by  MACOM 
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These  analyses  revealed  that  the  CPUU  has  risen  over  the  six-year  period 
studied.  Additionally,  comparing  the  baseline  years,  2004-2005,  to  the  post-ILP 
years,  2006-2009,  we  noticed  the  baseline  years’  CPUU  values  are  consistently 
lower. 


4.  Statistical  Testing 

It  is  important  that  we  point  out  that  for  this  study  we  had  the  entire 
population  of  Army  HMMWV’s  for  which  data  was  available.  Therefore,  strictly 
speaking,  there  is  no  call  for  the  testing  of  statistical  hypotheses,  in  which  data 
from  a  sample  is  used  to  draw  inferences  about  parameters  of  a  population. 
However,  it  may  be  reasonable  to  consider  the  set  of  yearly  HMMWV 
observations  as  if  they  were  a  sample  from  a  larger,  hypothetical  population  of 
possible  observations.  In  this  way,  the  observed  data  can  serve  as  a  sample 
from  this  larger  population,  which  may  represent  future  observations  on  these  (or 
other)  vehicles,  and  hypotheses  about  this  population  (and  in  particular  whether  it 
has  changed)  can  be  tested. 

In  order  to  have  a  more  robust  test,  we  looked  at  only  those  vehicles 
containing  values  for  the  CPUU  in  both  time  periods.  Specifically,  we  looked  at 
two-year  comparisons.  There  were  eight  different  combinations  of  two-year 
comparisons.  Vehicles  that  had  zero  usage  for  a  given  year  but  had  associated 
costs  for  that  same  year,  had  a  N/A  value  for  CPUU.  We  eliminated  those 
vehicles  that  had  a  N/A  for  the  CPUU  value  in  either  year.  We  then  conducted 
the  t-test  on  the  difference  of  the  log  of  the  CPUU  for  both  time  periods.  The  log 
was  used  because  the  assumptions  of  Normality  required  for  the  t-test  was  not 
plausible  for  the  original  data.  However,  after  transforming  the  data  using  the 
logarithmic  function,  the  data  appears  Normal  for  both  models.  This  is  displayed 
in  Figure  19  for  the  M998  vehicle.  While  not  every  year  combination  for  each 
vehicle  is  shown,  each  of  the  plots  are  similar  to  those  in  Figure  19. 
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Figure  19:  Histogram  of  M998  Log  of  CPUU  Difference  (2004  vs. 

Post-ILP  Years) 
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We  also  conducted  the  Wilcoxon  test.  For  this  test,  the  null  hypothesis  is 
the  same  as  the  t-test:  the  average  change  in  CPUU  between  the  two  years 
being  compared  is  equal  to  zero.  We  conclude  from  this  test  that  the  difference 
in  means  is  different  and  that  the  pre-ILP  years  are  cheaper.  Additionally,  we 
conducted  the  Chi-Squared  test  to  ensure  consistency  in  our  results.  For  the 
Chi-Squared  test,  the  null  hypothesis  is  that  the  percentage  for  which  the 
difference  between  CPUU  values  is  less  than  zero  is  equal  to  50%  (there  are  as 
many  vehicles  that  had  a  higher  CPUU  Pre-ILP  as  there  were  that  had  a  higher 
CPUU  Post-ILP).  The  final  column  in  Table  6  and  Table  7,  True-p,  is  the 
percentage  of  vehicles  that  had  a  higher  Pre-ILP  CPUU  than  the  Post-ILP  CPUU. 
For  both  models  in  nearly  every  instance,  this  value  is  less  than  50%,  which 
indicates  that  in  most  cases,  the  CPUU  was  lower  prior  to  implementation  of  the 
ILP.  These  results  confirm  that  on  average  the  Pre-ILP  CPUU  is  lower  than  the 
Post-ILP  CPUU.  Table  6  and  Table  7  contain  the  results  of  all  three  tests  for 
both  models. 
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T-Test  Wilcoxon  Chi-Squared 


Ha  =  Difference  in  means  is  NOT  zero 

Ha  =  Difference  in  means  is  NOT  zero 

Ha  =  True  p  is  not  equal  to  0.5 

Log) 

pld-New) 

Old-New 

Old>New 

T-stat 

Df 

P-value 

Result 

V 

P-Value 

Result 

ChiA2 

P-value 

True-p 

04  vs  06 

-9.96 

2468 

2.20E-16 

Reject 

1139967 

2.20E-16 

Reject 

64.80 

8.28E-16 

0.42 

04  vs  07 

-10.43 

1581 

2.20E-16 

Reject 

414444 

2.20E-16 

Reject 

109.92 

2.20E-16 

0.37 

04  vs  08 

-14.87 

1502 

2.20E-16 

Reject 

307282 

2.20E-16 

Reject 

196.90 

2.20E-16 

0.31 

04  vs  09 

-15.50 

1114 

2.20E-16 

Reject 

168809 

2.20E-16 

Reject 

181.61 

2.20E-16 

0.30 

05  vs  06 

0.55 

4404 

5.84E-01 

Fail 

4883129 

2.92E-01 

Fail 

2.01 

1.57E-01 

0.51 

05  vs  07 

-1.31 

3030 

1.91E-01 

Fail 

2112927 

4.09E-04 

Reject 

11.17 

8.31E-04 

0.47 

05  vs  08 

-10.83 

2646 

2.20E-16 

Reject 

1354358 

2.20E-16 

Reject 

104.53 

2.20E-16 

0.40 

05  vs  09 

-7.43 

1606 

1.70E-13 

Reject 

515748 

2.50E-12 

Reject 

49.49 

1.99E-12 

0.41 

Table  6:  M998  Two-Year  Comparison  Test  Results 


T-Test  Wilcoxon  Chi-Squared 


Ha  =  Difference  in  means  is  NOT  zero 

Ha  =  Difference  in  means  is  NOT  zero 

Ha  =  True  p  is  not  equal  to  0.5 

Log(Old-New) 

Old-New 

Old>New 

T-stat 

Df 

P-value 

Result 

V 

P-Value 

Result 

ChiA2 

P-value 

True-p 

04  vs  06 

-7.45 

178 

3.80E-12 

Reject 

2874 

8.50E-14 

Reject 

41.31 

1.29E-10 

0.26 

04  vs  07 

-7.26 

101 

8.22E-11 

Reject 

630 

2.68E-11 

Reject 

38.91 

4.34E-10 

0.19 

04  vs  08 

-7.37 

106 

4.02E-11 

Reject 

888 

5.05E-10 

Reject 

23.36 

1.34E-06 

0.26 

04  vs  09 

-5.33 

64 

1.35E-06 

Reject 

382 

6.51E-06 

Reject 

13.85 

1.98E-04 

0.26 

05  vs  06 

0.22 

333 

8.30E-01 

Fail 

26001 

3.50E-01 

Fail 

0.00 

1.00E+00 

0.50 

05  vs  07 

-0.19 

198 

6.20E-02 

Fail 

7655 

8.80E-03 

Reject 

8.04 

4.57E-03 

0.40 

05  vs  08 

-4.25 

182 

3.43E-05 

Reject 

5344 

1.84E-05 

Reject 

10.58 

1.14E-03 

0.37 

05  vs  09 

-3.44 

129 

7.93E-04 

Reject 

2672 

2.30E-04 

Reject 

11.70 

6.25E-04 

0.35 

Table  7:  M1097  Two-Year  Comparison  Test  Results 
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5. 


Mixed  Effects  Model 


The  final  piece  of  our  analysis  involved  using  a  mixed  effects  ANOVA 
model.  (Galwey,  2006)  In  this  model,  the  response  is  CPUU.  The  effect  of  year 
is  “fixed”  in  that  the  levels  of  year  are  known  in  advance,  but  the  effect  of  vehicle 
is  “random”  in  that  we  envision  (at  least  hypothetically)  the  observed  vehicles 
acting  as  a  random  sample  from  a  larger  population  of  possible  vehicles.  This 
approach  allows  us  to  consider  every  observation  for  every  vehicle,  instead  of 
simply  comparing  years  two  at  a  time,  while  adjusting  for  variability  among 
vehicles.  As  with  the  statistical  testing,  we  used  the  log  of  the  CPUU  because 
the  assumption  that  the  errors  are  Normal  was  not  plausible  for  the  original  data, 
but  for  the  log,  this  assumption  was  valid. 

Specifically,  the  model  proposes  that 

Cij=  jj  +  yj  +  Vi  +  e-j, 

where 

jj  =  intercept, 

Yj  =  jth  year  effect  (fixed), 

Vi  =  ith  vehicle  effect  (random,  assumed  N(0,  ov2),  and 

Cjj  =  error  (assumed  iid  N(0,  oe2)) 

The  null  hypothesis  of  interest  is  that  the  year  effects  are  identically  zero  (that  is, 
that  overall  average  CPUU  has  remained  constant  across  years).  This  model 
was  implemented  using  the  Ime4  package  in  R.  (Bates,  2010)  That  package 
does  not  directly  provide  p-values  for  the  F  test  associated  with  that  null 
hypothesis,  but  it  is  possible  to  compare  two  nested  models.  We  compared  the 
above  model  to  a  similar  model  that  included  vehicle  but  excluded  year.  The 
resulting  p-value  was  smaller  than  .01,  leading  us  to  reject  the  null  hypothesis  of 
constant  average  CPUU  and  to  conclude  that  the  average  CPUU  value  has 
changed  with  time.  The  same  conclusion  is  reached  when  comparing  the  two 
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models’  AIC  values.  Since  the  model  that  included  the  year  had  a  smaller  (that  is, 
better)  AIC  than  the  one  without,  we  reject  the  null  hypothesis. 

To  examine  the  effect  of  the  ILP  more  directly,  we  built  a  third  model  in 
which  the  year  was  represented  by  a  two-level  factor  denoting  the  pre-ILP  (2004- 
2005)  and  post-ILP  (2006-2009)  years.  We  can  then  examine  the  null  hypothesis 
of  no  effect  of  the  ILP.  In  the  comparison  of  this  model  to  the  baseline  (vehicle- 
only)  model,  we  observe  a  slightly  smaller  AIC,  and  the  p-value  associated  with 
that  null  hypothesis  was  less  than  .01,  suggesting  that  the  imposition  of  the  ILP 
had  a  significant  effect  on  average  CPUUs.  Based  on  our  previous  analysis  and 
this  test,  it  is  safe  to  conclude  that  the  ILP  is  not  providing  any  cost  savings  to  the 
DoD. 

B.  OPERATIONAL  AVAILABILITY 

The  availability  data  collected  from  LIDB  was  not  broken  down  by  vehicle 
like  the  cost  and  usage  data.  Instead,  it  was  broken  down  by  Unit  Identification 
Number  (UIC),  Location,  and  MACOM.  Each  UIC  reported  a  Mission  Capable 
(MC)  percentage  on  the  15th  of  every  month.  This  is  the  metric  we  used  for  this 
study. 

First,  we  looked  at  the  Operational  Availability  versus  the  calendar  quarter. 
This  showed  a  fairly  constant  rate  until  the  latter  part  of  2007  when  there  were 
noticeable  reductions  in  the  Operational  Availability  rate.  For  the  M998  vehicle, 
we  see  a  slight  increase  in  readiness  until  that  point  in  2007  when  the  rate  drops 
about  2%.  The  Ml 097  falls  10%  between  the  2nd  quarter  of  FY2007  and  the 
2nd  quarter  of  FY2008.  They  both  appear  to  level  out  near  89-90%  by  the  end  of 
2009.  It  is  clear  that  something  occurred  in  2007  to  cause  a  drop  in  readiness.  It 
is  also  clear  that  the  Operational  Availability  rates  prior  to  the  ILP  implementation 
were  higher  than  those  after.  These  results  are  displayed  in  Figure  20.  The 
surge  in  Iraq  occurred  in  2007,  and  this  could  have  had  some  effect  on  this 
metric.  Also,  as  documented  in  Chapter  III,  the  number  of  Ml 097  vehicles 
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increased  between  2006  and  2009.  An  increase  in  vehicles  without  an  increase 
in  logistical  support  could  have  driven  the  average  Operational  Availability  down. 


Mean  Availability  by  Quarter 


Figure  20:  Mean  Operational  Availability  of  Vehicles  by  Quarter 

As  in  the  CPUU  analysis,  we  looked  at  the  Operational  Availability  by 
MACOM  before  and  after  the  implementation  of  the  ILP.  The  results  were  not  as 
conclusive  as  with  the  CPUU.  While  the  post-ILP  Operational  Availability 
numbers  were  on  average  lower  than  those  from  before,  Operational  Availability 
for  the  two  MACOM’s  that  own  the  majority  of  the  vehicles  remained  fairly 
constant.  In  the  case  of  the  M998  at  FORSCOM  there  was  a  3%  increase, 
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indicating  an  improvement  since  the  ILP  implementation.  But,  Army  Materiel 
Command  (USAMC)  had  large  drops  in  Operational  Availability  for  both  vehicle 
models.  Considering  that  AMC  owned  less  the  10%  of  the  Ml 097s  and  less  than 
1%  of  the  M998s,  this  large  drop  does  not  lead  us  to  any  conclusions.  A 
summary  of  the  results  are  presented  in  Table  8. 


MACOM 

M998 

M1097 

<=2005 

>2005 

Score 

<=2005 

>2005 

Score 

EUSA 

92.84% 

94.96% 

1 

94.54% 

92.68% 

0 

FORSCOM 

89.62% 

92.49% 

1 

95.22% 

96.53% 

1 

IMCOM 

NA 

94.35% 

NA 

NA 

99.86% 

NA 

MDW 

93.67% 

99.31% 

1 

98.41% 

99.50% 

1 

NETCOM 

94.60% 

94.32% 

0 

97.67% 

96.50% 

0 

NGB 

89.90% 

90.77% 

1 

95.54% 

95.43% 

0 

OSD 

97.82% 

97.21% 

0 

NA 

100.00% 

NA 

USACIDC 

94.65% 

96.10% 

1 

100.00% 

99.59% 

0 

USAIMA 

84.25% 

100.00% 

1 

97.05% 

99.99% 

1 

USAMC 

88.95% 

65.31% 

0 

83.24% 

69.09% 

0 

USARC 

NA 

90.43% 

NA 

NA 

95.47% 

NA 

USARCENT 

NA 

95.72% 

NA 

NA 

94.03% 

NA 

USAREUR 

7A 

92.13% 

89.52% 

0 

94.34% 

97.17% 

1 

USARPAC 

92.43% 

93.18% 

1 

95.01% 

90.69% 

0 

USARSO 

6A 

NA 

93.25% 

NA 

NA 

97.08% 

NA 

USASOC 

96.67% 

97.48% 

1 

98.28% 

99.08% 

1 

USINSCOM 

93.62% 

94.61% 

1 

94.30% 

95.85% 

1 

Averages 

92.39% 

92.88% 

95.30% 

95.21% 

Final  Score  Percentage 

0.67 

0.55 

Table  8:  Operational  Availability  Comparison  by  MACOM 


Histograms  for  the  Operational  Availability  of  the  M998  vehicle  before  and 
after  implementation  of  the  ILP  are  displayed  in  Figure  21.  Plots  of  the  M1097, 
while  not  shown  here,  displayed  similar  results.  Based  on  these  plots  and  the 
results  from  above  across  MACOMs,  the  Operational  Availability  does  not  appear 
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to  change  in  any  significant  manner.  In  fact,  it  appears  to  hold  relatively  constant 
between  the  90-95%  range.  This  has  been  the  DoD  unwritten  standard  for 
weapons  systems  Operational  Availability. 


histogram  of  Pre-HP  Availability  for  M998 
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Figure  21 :  Comparison  Histogram  of  Operational  Availability  for 

M998  Vehicle 
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C.  CONCLUSION 

This  study  analyzed  two  of  the  DoD-defined  metrics  to  determine  if  the 
HMMWV  ILP  was  functioning  better  than  the  traditional  maintenance  plan. 
Several  different  approaches  were  taken  to  view  the  performance  of  the  program 
using  the  CPUU  and  Operational  Availability.  There  appeared  to  be  an  increase 
in  cost  and  no  significant  change  in  Operational  Availability,  over  the  six-year 
period  studied.  Hypothesis  testing  was  conducted  on  the  metrics  for  the  two  time 
periods,  and  it  revealed  that  there  was,  indeed,  a  statistically  significant 
difference  between  the  two  time  periods  for  CPUU.  To  further  confirm  this,  a 
year-to-year  comparison  was  conducted  as  well  and  the  results  were  confirmed. 
A  mixed  effects  model  was  built,  and  it  was  determined  that  the  year  had  a 
significant  relationship  with  the  CPUU.  On  average,  the  CPUU  prior  to 
implementation  of  the  ILP  is  lower  than  the  CPUU  after;  however,  it  cannot  be 
determined  that  the  ILP  is  the  cause  of  this  decrease.  The  only  conclusive 
statement  that  can  be  made  is  that  the  ILP  has  not  provided  the  DoD  with  cost 
savings. 

The  accuracy  of  the  data  was  an  area  of  concern  for  the  writer  and  much 
time  and  effort  was  exhausted  to  correct  it.  However,  there  is  still  doubt  as  to  its 
complete  validity.  Since  the  metric  studied  was  based  on  the  usage  data,  there 
could  be  inherent  errors  in  these  results. 
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V.  SUMMARY  AND  RECOMMENDATIONS 


The  Department  of  Defense  has  claimed  that  Performance-Based 
Logistics  is  the  preferred  method  of  Life-Cycle  Support  for  weapons  systems. 
This  approach  was  implemented  in  an  effort  to  increase  availability  while 
decreasing  costs.  Unfortunately,  based  on  the  results  found  in  this  study,  neither 
of  these  results  appears  to  have  occurred  in  the  case  of  the  Army  HMMWV  ILP. 

Four  principal  conclusions  were  found  in  the  course  of  this  study: 

1 .  CPUU  had  an  upward  trend  from  2004  to  2009; 

2.  CPUU  was  lower  before  the  ILP  and  higher  after  the  ILP 
implementation; 

3.  Operational  Availability  showed  no  significant  trend  from  2004  to 
2009;  and 

4.  Operational  Availability  experienced  no  significant  change  due  to 
the  ILP. 

The  CPUU  rose  continuously  over  the  time  period  studied.  While  there 
were  some  drops  in  certain  quarters,  the  overall  trend  was  upward.  While 
unadjusted  costs  are  expected  to  grow  over  time,  this  study  accounted  for 
inflation  and  still  a  rise  in  costs  was  observed.  Additionally,  when  the  two  time 
periods  were  compared,  the  time  period  before  the  ILP  was  determined  to  have  a 
smaller  CPUU  than  the  post-ILP  time  period.  This  was  found  through  several 
different  analyses  and  confirmed  using  statistical  hypothesis  testing.  Again,  we 
point  out  that  the  CPUU  is  driven  by  the  usage  data,  which  we  found  to  be  a 
major  area  of  concern  in  this  study.  However,  every  effort  was  made  to  use  that 
data  that  was  available  and  correct  obvious  errors  in  a  logical  manner.  The 
CPUU  rise  was  subtle  and  could  be  attributed  to  rise  in  costs  over  time.  It  could 
also  be  attributed  to  the  fact  that  usage  went  down  over  this  same  time  period.  If 
there  are  time-related  maintenance  requirements,  then  the  CPUU  would  be 
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driven  up  because  the  maintenance  is  still  conducted  regardless  of  the  usage. 
The  costs  of  the  HMMWV  program  could  have  seen  a  similar  (or  even  greater) 
rise  even  if  the  ILP  had  not  been  in  place. 

While  the  CPUU  rose  over  the  study  period,  the  Operational  Availability 
did  not  show  any  change.  While  there  appears  to  be  a  slight  drop  in  the  latter 
portion  of  2007,  this  decrease  was  minute  and  the  overall  Operational  Availability 
remained  above  90%  throughout  the  period  studied.  The  slight  drop  could  have 
been  attributed  to  other  things  such  as,  perhaps,  the  surge  in  Iraq  in  2007.  This 
metric  remained  relatively  constant  for  both  models  studied  until  that  time  frame, 
at  which  point  a  decrease  was  seen.  Even  after  the  decrease,  both  models  were 
maintaining  nearly  90%  Operational  Availability,  which  is  the  unwritten  minimum 
throughout  the  DoD.  While  the  DoD  wants  to  achieve  cost  savings,  as  well  as 
availability  increases,  the  availability  metric  is  of  higher  importance.  Operational 
Availability  is  the  key  for  our  warfighters  to  be  able  to  conduct  their  missions 
successfully.  Regardless  of  costs,  if  weapons  systems  are  not  available  for  use, 
we  will  have  many  things  to  be  concerned  about  other  than  costs. 

While  these  conclusions  are  important,  the  primary  finding  in  this  study 
was  that  the  data  to  conduct  an  analysis  of  PBLs  is  simply  not  accurate  enough. 
In  order  to  give  decisions  makers  a  tool  to  plan  for  future  Life-Cycle  Sustainment 
of  weapons  systems,  analysts  need  sufficient  and  accurate  data  that  can  be  used 
to  analyze  the  DoD  prescribed  metrics.  The  key  problem  encountered  in  this 
study  was  the  usage  data  obtained  from  the  Army  LIDB  database.  Missing 
values  and  obvious  errors  were  rampant  in  the  data.  While  we  went  to  great 
efforts  to  correct  these  errors,  this  could  still  have  an  effect  on  the  outcome. 

The  Army  LIDB  is  an  improvement  from  the  past  and  has  the  capability  to 

provide  all  the  necessary  data  for  analysts  to  conduct  studies  such  as  this. 

However,  the  inputs  into  this  database  are  clearly  in  need  of  refinement.  Most 

likely,  these  errors  are  being  committed  at  the  lowest  level,  which  is  when  the 

maintenance  man  goes  out  to  take  the  monthly  reading  and  enters  it  into  the 

computer.  There  needs  to  be  more  quality  control  on  this  process.  The  best 
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solution  would  be  for  the  Army  to  develop  electronic  tracking  of  the  data,  and 
have  a  system  similar  to  the  Navy’s  remote  monitoring  system  being  installed  on 
ships.  This  system  allows  the  watchman  to  simply  plug  in  a  hand-held  device  to 
a  piece  of  equipment  and  the  operating  parameters  are  automatically  uploaded. 
The  Army  could  develop  a  similar  device  such  that  each  time  it  is  plugged  in  to  a 
vehicle,  the  current  odometer  reading  is  uploaded,  thus  preventing  any  human 
error. 

The  original  plan  for  this  study  was  to  use  the  LSD-41/49  Class  diesel 
engines,  as  well  as  the  HMMWV,  to  examine  the  effect  of  ILP.  The  Navy  made  a 
shift  in  diesel  maintenance  practices  on  the  LSD-41/49  Class  ship  in  the  past 
decade  after  one  ship  failed  to  deploy  and  three  other  ships  deployed  but  were 
essentially  ineffective.  (Caccese,  2007,  p.  4)  The  basic  idea  behind  this  new 
practice,  Diesel  Readiness  System,  was  to  increase  Operational  Availability  and 
reduce  costs.  These  goals  would  be  accomplished  through  a  one-time  base  line 
assessment,  followed  by  more  proactive  (vice  reactive)  maintenance  schedules. 
Some  of  the  major  changes  would  include  automated  collection  of  machinery 
history  logs,  standardized  repair  requirements,  new  failure  and  costs  metrics,  off- 
ship  lube  oil  analysis,  and  program  management  for  business  rules.  (Caccese, 
2007,  p.  6) 

While  the  DRS  program  has  made  improvements  over  the  previous  diesel 
maintenance  program,  it  would  be  interesting  to  see  if  a  Performance-Based 
Logistics  (PBL)  contract  with  the  Original  Equipment  Manufacturer  (OEM)  would 
produce  even  more  costs  savings  for  the  Navy.  Fairbanks  Morris  Corporation 
(FMC)  and  Northrop  Grumman  have  offered  to  establish  such  a  partnership  for 
the  LSD  class  ship.  The  offer  has  never  come  to  fruition;  however,  FMC  was 
contacted  and  asked  about  a  possible  maintenance  plan  for  these  engines.  FMC 
was  kind  enough  to  release  some  figures  on  what  they  would  have  charged  the 
Navy  for  a  30-year  maintenance  program,  which  would  require  the  engines  to  be 
maintained  at  90%  Operational  Availability  or  better. 


59 


One  of  the  key  topics  of  concern  amongst  Navy  brass  is  Total  Ownership 
Costs  (TOC)  for  diesel  engines.  In  our  study,  the  TOC  was  going  to  be  the 
metric  of  interest  for  the  LSD  41/49  Class  ship.  Unfortunately,  there  was  not 
enough  data  or  historical  information  to  accurately  calculate  the  TOC  in  the  time 
allotted.  Therefore,  the  diesel  engine  was  abandoned  and  the  study  was 
narrowed  down  to  the  Army  HMMWV  only. 

While  the  Navy  is  developing  monitoring  systems,  as  mentioned 
previously,  its  database  is  years  behind  that  of  the  Army’s.  The  Navy  and  Marine 
Corp  Visibility  and  Management  of  Operating  and  Support  Costs  (VAMOSC) 
database  contains  a  myriad  of  data,  but  gives  summary  statistics  instead  of 
weapon  system  level  data.  It  is  important  for  the  analysts  to  have  data  that  is  at 
the  weapon  system  level.  Furthermore,  VAMOSC  is  difficult  to  use.  The  queries 
are  not  intuitive  and  have  too  many  options  that  leave  the  user  unsure  about  the 
type  of  report  he  or  she  is  producing.  Additionally,  it  is  difficult  to  export  the  data 
into  a  Comma  Separated  Value  (CSV)  or  some  other  format  that  is  commonly 
used  by  analysts.  So,  the  Army  and  Navy  need  to  collaborate  and  share  their 
strengths  when  it  comes  to  data  collection  and  storage.  This  would  provide 
analysts  with  accurate  data  that  in  turn  could  provide  decision  makers  sound 
decision-making  tools  for  future  planning  of  weapons  systems  life-cycle 
sustainment.  That  the  data  was  dirty  and  missing  came  as  no  surprise,  as  many 
of  the  studies  reviewed  had  similar  findings,  as  documented  in  Chapter  II. 

There  are  ample  opportunities  for  future  studies  on  Performance  Based 
Logistics.  The  DoD  has  prescribed  five  metrics:  1)  Operational  Availability,  2) 
Operational  Reliability,  3)  Cost  per  Unit  Usage,  4)  Logistics  Footprint,  and  5) 
Logistics  Response  Time.  This  thesis  addressed  Operational  Availability  and 
CPUU,  but  follow  on  studies  could  address  the  other  metrics.  Furthermore,  other 
platforms  under  PBL  contracts  could  be  studied  using  the  same  metrics,  any 
combination  of  the  others,  or  all  five  of  them. 
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APPENDIX:  DATA  CORRECTION 


This  appendix  discusses  various  refinements  we  made  to  improve  the 
quality  of  the  data  for  our  analysis. 

A.  SERIAL  NUMBERS 

1.  WON  Data 

The  serial  number  data  for  several  WONs  from  LIDB  required  some 
correction.  Based  on  the  RAND  study,  the  data  from  LIDB,  and  emails  from 
Army  Materiel  Command,  it  was  determined  that  a  valid  serial  number  was  a 
three-  to  six-digit  number  (Neal,  2010;  RAND  Corporation,  2008,  56).  Originally, 
this  data  contained  88,036  unique  serial  numbers  from  264,554  WONs.  In  our 
data,  2,092  serial  numbers  were  entered  along  with  their  corresponding 
registration  number  separated  by  either  a  slash  or  a  dash.  The  registration 
number  was  removed  from  those  entries.  There  were  582  entries  that  contained 
only  the  registration  number.  We  replaced  this  with  the  corresponding  serial 
number  for  each  vehicle.  There  were  2,116  entries  which  had  a  valid  six-digit 
number  either  preceded  or  followed  by  another  character.  These  characters 
were  removed  leaving  only  the  six-digit  number.  We  removed  20,513  entries 
because  the  serial  number  entry  was  unable  to  be  repaired  or  it  had  not  been 
entered.  After  these  repairs  and  eliminations,  72,394  unique  serial  numbers 
remained  for  analysis. 

2.  Usage  Data 

The  serial  number  entries  in  the  usage  data  gathered  from  LIDB  was  in 
great  condition  and  only  required  a  few  eliminations.  For  the  M998  model,  the 
original  data  had  566,240  line  items  from  the  years  2000  to  2009.  From  these 
line  items  we  found  48,266  unique  serial  numbers.  We  eliminated  204  of  these 
entries  due  to  registration  numbers  not  corresponding  to  serial  numbers.  For 
example,  the  registration  number  was  recorded  as  being  the  same  for  multiple 
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serial  numbers.  We  were  left  with  48,062  serial  numbers  for  the  M998  model. 
For  the  Ml 097  model,  the  exact  same  process  for  elimination  was  used  resulting 
in  an  original  unique  set  of  serial  numbers  of  25,349  being  reduced  to  25,055. 

3.  Combining  the  Serial  Number  Data 

Using  the  serial  number  lists  created  from  each  of  the  two  data  sets,  we 
created  a  master  list.  The  two  models  combined  had  72,394  serial  numbers  from 
the  WON  data  and  73,117  from  the  usage  data.  A  total  of  52,147  numbers 
appeared  in  both  lists.  Of  that,  35,590  were  M998s  and  16,584  were  Ml 097s. 
This  resulted  in  142,186  WONs  for  M998  and  46,430  WONs  for  M1097.  From 
the  original  WON  data,  we  eliminated  29%  of  the  line  items  from  the  repair, 
elimination,  and  matching  of  the  serial  numbers  across  both  data  sets. 

B.  USAGE  DATA 

The  vehicle  odometer  data  gathered  from  LIDB  had  a  large  number  of 
missing  values  and  other  errors,  such  as  double  entries  that  did  not  correspond, 
and  unusually  high  readings  from  one  month  to  the  next.  First,  any  values  that 
were  over  300,000  were  removed.  The  data  was  then  cut  into  50,000  mile 
categories  across  all  the  years,  2000-2009,  for  each  vehicle.  Each  reading  was 
assigned  a  category  between  one  and  six  (0-50,000=1, 50,001-100,000=2,  etc.). 
Next,  the  mode  of  these  categories  for  each  individual  vehicle  was  determined. 
Values  that  fell  in  the  mode  of  the  categories  plus  or  minus  one  increment  were 
kept.  For  example,  if  a  vehicle  had  a  mode  of  three,  then  all  values  that  were 
assigned  a  category  of  two,  three,  or  four  were  retained.  All  others  were  marked 
as  “N/A.”  This  same  process  was  repeated  for  each  year  with  the  cuts  being 
reduced  to  8,000.  The  process  was  repeated  once  more  for  each  quarter  using  a 
3,000  mile  cut.  This  process  eliminated  the  cases  where  we  had  anomalies  in 
the  month-to-month  readings.  In  cases  where  there  were  multiple  entries  for  one 
vehicle  for  a  single  year,  the  entries  were  merged.  If  one  entry  had  a  value  for  a 
particular  month  and  the  other  entry  did  not,  then  the  value  was  retained.  If  there 
were  multiple  different  entries  for  a  single  month,  then  the  maximum  entry  was 
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retained.  Next,  we  looked  through  each  reading  for  a  single  vehicle  and  if  the 
previous  months  reading  was  higher  than  the  current  months  reading,  the  current 
months  reading  was  replaced  with  “N/A.”  We  calculated  the  actual  usage  by 
subtracting  the  n+1  month  from  each  month  assuming  that  the  readings  were 
taken  on  the  final  day  of  each  month.  Because  of  the  eliminations,  we  used 
simple  imputation  to  fill  in  gaps  in  the  data.  Then,  the  sum  of  the  monthly  data 
was  calculated  for  each  vehicle  for  each  year. 

1.  Cost 

The  cost  data  was  associated  with  each  individual  WON,  and  therefore, 
needed  to  be  summed  in  order  to  determine  total  costs  for  a  vehicle  for  each 
year  studied.  Using  the  remaining  WON  data  after  the  serial  number  corrections, 
we  built  a  new  database  that  contained  the  sum  of  the  costs  for  each  vehicle  for 
each  year.  We  also  counted  the  number  of  WONs  for  a  particular  vehicle  for 
each  year  and  included  that  information  in  the  new  database.  The  primary  metric 
studied,  CPUU,  was  calculated  by  dividing  the  total  cost  for  a  vehicle  for  the  year 
by  the  total  usage  for  that  year.  These  values  were  then  adjusted  for  inflation 
and  converted  to  FY1 1$.  The  Operations  and  Maintenance  Army  Appropriations 
index  was  used  for  this  adjustment.  (Naval  Center  for  Cost  Analysis,  2010) 
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